Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exists data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values--an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own prediction is a sufficient condition for alignment. Experiments on a real AI-assisted decision making scenario where a classifier provides decision support to human decision makers validate our theoretical results and suggest that alignment may lead to better decisions.

Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions, and utilized these models to improve human-AI team performance. Meanwhile, due to the black-box'' nature of AI models, providing AI explanations to human decision makers to help them rely on AI recommendations more appropriately has become a common practice. In this paper, we explore whether we can quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign.

High stakes decision-making often requires a continuous interplay between evolving evidence and shifting hypotheses, a dynamic that is not well supported by current AI decision support systems. In this paper, we introduce a mixed-initiative framework for AI assisted decision making that is grounded in the data-frame theory of sensemaking and the evaluative AI paradigm. Our approach enables both humans and AI to collaboratively construct, validate, and adapt hypotheses. We demonstrate our framework with an AI-assisted skin cancer diagnosis prototype that leverages a concept bottleneck model to facilitate interpretable interactions and dynamic updates to diagnostic hypotheses.

AI-assisted decision making becomes increasingly prevalent, yet individuals often fail to utilize AI-based decision aids appropriately especially when the AI explanations are absent, potentially as they do not %understand reflect on AI's decision recommendations critically. Large language models (LLMs), with their exceptional conversational and analytical capabilities, present great opportunities to enhance AI-assisted decision making in the absence of AI explanations by providing natural-language-based analysis of AI's decision recommendation, e.g., how each feature of a decision making task might contribute to the AI recommendation. In this paper, via a randomized experiment, we first show that presenting LLM-powered analysis of each task feature, either sequentially or concurrently, does not significantly improve people's AI-assisted decision performance. To enable decision makers to better leverage LLM-powered analysis, we then propose an algorithmic framework to characterize the effects of LLM-powered analysis on human decisions and dynamically decide which analysis to present. Our evaluation with human subjects shows that this approach effectively improves decision makers' appropriate reliance on AI in AI-assisted decision making.

Whenever an AI model is used to predict a relevant (binary) outcome in AI-assisted decision making, it is widely agreed that, together with each prediction, the model should provide an AI confidence value. However, it has been unclear why decision makers have often difficulties to develop a good sense on when to trust a prediction using AI confidence values. Very recently, Corvelo Benz and Gomez Rodriguez have argued that, for rational decision makers, the utility of AI-assisted decision making is inherently bounded by the degree of alignment between the AI confidence values and the decision maker's confidence on their own predictions. In this work, we empirically investigate to what extent the degree of alignment actually influences the utility of AI-assisted decision making. To this end, we design and run a large-scale human subject study (n=703) where participants solve a simple decision making task - an online card game - assisted by an AI model with a steerable degree of alignment. Our results show a positive association between the degree of alignment and the utility of AI-assisted decision making. In addition, our results also show that post-processing the AI confidence values to achieve multicalibration with respect to the participants' confidence on their own predictions increases both the degree of alignment and the utility of AI-assisted decision making.

The potential is not necessarily realized, however, because of several challenges: debates on ethical resposibility of decisions [8, 26, 44], the human ability to recognize when AI advice should be taken [43], mental models (biases) regarding AI performance and ability [12, 27] to perform well on subjective tasks, and effects of how the AI advice is delivered [46]. Many research directions thus aim to resolve these barriers to complementarity in human-AI performance, including examining the effects of having AI systems explain predictions [4] using explainable AI (XAI) methods, introducing cognitive forcing functions when presenting AI advice [6], adjusting AI advice interactions/presentation methods [40], and adjusting task framing to account for mental models about the types of tasks AI can work with [9]. In AI-assisted decision making, the human makes the final decision, bearing full responsibility for its consequences. Performance pressure from responsibility can influence decision making behavior [2]. The bulk of research working towards complementary human-AI performance isolates human behavior away from the effects of performance pressure because the field is rapidly evolving its understanding of how humans perceive and work with AI tools. Intrinsically high and low stakes tasks are used in these experiments, but the stakes have little tangible effect or implication for evaluators. Hence, we observe a gap in the literature of how people rely on AI assistants under performance pressure, or when stakes matter personally. In this work, we seek to understand how performance pressure affects AI advice usage when AI advice is provided as a second opinion. We induce performance pressure through a pay-by-performance scheme framed as a loss.

Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values.

Artificial Intelligence (AI) is increasingly employed in various decisionmaking However, empirical research reveals several limitations within tasks, typically as a Recommender, providing recommendations the existing AI-assisted decision-making framework, wherein AI that the AI deems correct. However, recent studies suggest this acts primarily as a recommender. One notable issue is that individuals, may diminish human analytical thinking and lead to humans' inappropriate when passively receiving AI suggestions, seldom engage reliance on AI, impairing the synergy in human-AI teams. in analytical thinking [3, 7, 38]. Furthermore, people frequently In contrast, human advisors in group decision-making perform inappropriately rely on the AI's recommendations (such as overreliance various roles, such as analyzing alternative options or criticizing and under-reliance) [8, 30, 33, 46] and the mere provision of decision-makers to encourage their critical thinking. This diversity AI explanations can, paradoxically, exacerbate overreliance [2, 37]. of roles has not yet been empirically explored in AI assistance. In In comparison, in human-human decision-making, beyond recommenders, this paper, we examine three AI roles: Recommender, Analyzer, and human advisors sometimes play other types of roles, Devil's Advocate, and evaluate their effects across two AI performance such as helping the decision-makers analyze the pros and cons of levels. Our results show each role's distinct strengths and different alternatives instead of directly giving recommendations, or limitations in task performance, reliance appropriateness, and user critically challenging decision-makers' initial views [40, 42].